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ABSTRACT 

Linking teacher treatment to student impact is not as 
straightforward as it may seem initially. The difficulties in 
measuring the impact on students are illustrated in this discussion 
of the Ohio Statewide Systemic Initiative (SSl) . Since the inception 
of the SSI, the external evaluators had planned to use data from the 
state’s proficiency tests for ninth grade as one measure of student 
impact, but they encountered numerous difficulties, largely because 
the tests had not been completely developed. A pilot test of the SSI 
mathematics test in 1994 indicated the advisability of using a 
revised version of the pilot test in 1995. The revised Mathematics 
Discovery Test was administered to 1,070 students whose teachers had 
participated in the Project Discovery professional development for 
use of the SSI and 682 students whose teachers had not participated 
in Project Discovery. A Science Discovery Test was also administered 
to 1,127 Discovry students and 789 non~Dis covery students. Results of 
these two tests show the superior performance of the students of 
Discovery teachers. Experience in the test development process 
demonstrates the importance of teacher involvement in the development 
of tests designed to provide student impact data. Key stakeholders 
must be involved to ensure the appropriate alignment of the 
instrument. (Contains two figures and five tables.) (SLD) 
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Overview 



As the external evaluators of Ohio’s Statewide Systemic Initiative (SSI), we had readily 
obtained data from classroom observations, teacher surveys, and interviews with teachers and 
administrators that indicate that the SSI is having a positive impact on students; however, we 
realized that more systematic data on student performance would be an essential measure in 
demonstrating the project’s success. Upon encountering numerous obstacles in our efforts to 
utilize secondary data (collected by the state as part of its statewide testing program), we 
moved toward a process of creating mathematics and science tests m which skills expected to 
be imDacted by the SSI would be measured. Unplanned and unanticipated were the benefits 
to the SSI itself through the inclusion of numerous stakeholders from the state s education 
community in the test development process and the associated discussions of appropriate 

outcomes. 

There are currently 25 SSI states funded by the National Science Foundation and 
supplemented by various matching funds from each state. The SSI states were given the 
mission of implementing systemic reform in mathematics and science education across their 
entire state An SSI’s impact on students is, of course, one of the key pieces in determining 
the SSI’s success. For most SSIs, including the one covered in this report, the major 
treatment is professional development for teachers. 

Linking teacher treatment to student impact is not as straightforward as it may initially seem. 
Ouestions abound: what is the intensity of teacher treatment needed to have an impact on 
students 9 How long after the treatment would we expect to see changes in student skills 
associated with the professional development? What group of students would be an 
appropriate control group to the experimental SSI group? What test data are available Aa 
would provide an acceptable measure of student impact? Do we need to develop tests tha 
would specifically measure areas of change targeted by the SSI? If so, what are the essential 
criteria by which the design of such a test needs to be guided? 

Obstacles and Alternatives 

Ohio has a statewide testing system in which ninth graders are required to pass competency 
tests in mathematics, reading, and citizenship. A science section is m the process of being 
developed Since the onset of the SSI, we had planned to use data from the statewide ninth 
grade mathematics proficiency test as one measure of student impact. However, we 
encountered numerous obstacles in our efforts to use the statewide data. We were especially 
concerned about the availability of proficiency data in a timely fashion and of being able to 
obtain results on individual test items. Furthermore, in Ohio s SSI known as Project 
Discovery, "the teacher professional development program consisted of six week intensive 
"summer institutes" in inquiry instruction in one of three disciplines (mathematics, physics, 
and life science). Because the first step in Discovery was to prepare a cadre of people who 
could deliver these institutes, the number of participants in the first year (1992) was less than 
100 Although by the third summer (1994) the total number of teacher participants was 
approximately 750, the relatively small number of SSI teachers statewide complicated our 
ability to use secondary databases such as statewide proficiency test results At a March 1994 
evaluation meeting, project staff and the evaluators made a commitment to developing a 
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mathematics test that would allow analysis of student performance on specific test items and 
which could be administered as soon as possible. 

Earlier in the year, we had obtained the National Assessment of E ^ cat ion^ Progres S (NAEP) 
publicly released eighth grade mathematics and science test items (from NAEP tests 
administered in 1986, 1990, and 1992). In April 1994, the Ohio SSI formed a task force 
composed^ university-level mathematics faculty, members of the SSI s regional mathematics 
Academic Leadership Teams (composed of school teachers and;Sc,enti^Mafriemat,cian 
Educators"), and other SSI project staff. This task force drew from the NAEP items m 
developing an instrument specifically designed to measure students abilities in the 
mathematics and science content and higher-order thinking and problem solving j** s 
reflected in the inquiry and constructivist methods of teaching exemplified in Project 
Discover professional development naming. Because NAEP uses a complex matnx sampling 
of studems for which all items are not made publicly available, the Dtscovery test results 
cannot be benchmarked against National NAEP results. 

Pilot Test Results (Spring 1994) 

In late April through May 1994, project staff in two of the SSI regions asked SSI teachers 
and those selected for the upcoming 1994 summer institutes (the control ^ £ ^ h ere 
the SSI Mathematics Test to their eighth grade mathematics classes. A total of 2 1 tcaehcrs, 

10 SSI teachers and 1 1 non-SSI teachers, volunteered to administer the test to students. A 
total of 873 SSI students and 718 non-SSI students took the test. 

Although we recognize the limitations of the pilot testing research design (including the small 
sample of teachers and the non-random nature of the selection process) the pilot test e ort 
provided useful preliminary information. In the questionnaire accompanying the 
students confirmed that SSI teachers were using teaching practices modelled m the siunm 
institutes, such as cooperative groups. Furthermore, SSI students performance on the test 
was consistent with the prediction that inquiry-based instruction would have a positive 
influence on problem-solving and conceptual thinking skills. Finally, the pilot test resul 
convinced us of the value of implementing such a test in 1995 with a larger number of 

randomly selected teachers. 

Spring 1995 Results 

In conjunction with the evaluators, the mathematics test task force revised the pilot t test. .In an 
effort to make the assessment more "inquiry-oriented," the new test mstructed swdents to 
explain several of their answers. The 1995 Mathematics Discovery Test consisted of U items 
and the 1995 Discoverv Science Test contained 29 items, mchidmg multiple-choice items 
' ^ms involving tables and graphs, and open-ended questions, in addition we expanded I te*ng 
to a larger number of teachers and used a stratified random sample m risk™ 

SSI teachers. Since we were especially interested in the impact o e g 

students, we oversampled teachers in urban, high minority schools. 

In the spring of 1995, 2,197 students whose teachers underwent Project Discovery 
profoS development and 1,471 shidents whose teachers received no Project Discovery 
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treatment were tested with a battery of mathematics and science items developed by the 
National Assessment of Educational Progress (NAEP). 

Distinct patterns emerged about the students of teachers who participated in the Discovery 
program when compared to those who did not. These patterns were similar in both 
mathematics and science. First, on both close-ended and open-ended questions, Discovery 
students significantly outperformed non-Discovery students after controlling for sample 
differences. Second, except for boys in mathematics, all gender and ethnic groups whose 
teachers received the Discovery treatment performed significantly better than comparable 
students of teachers who did not. Third, student attitudes about their subject were similar for 
both Discovery students and non-Discovery students. Fourth, Discovery students reported 
participating in more hands-on and inquiry based activities than did non-Discovery students. 



The Teacher Sample 

The sample of teachers who were asked to give their students the Discovery Tests were 
selected randomly from both teachers who had participated in Project Discovery (the 
"treatment” group, also called Discovery teachers) and from those identified as the next cohort 
of Project Discovery, but not yet treated (the "control" group, also called non-Discovery 
teachers). The treatment sample was selected from those teachers who had received the 
Discovery professional development between 1992 and 1994. 

Before randomly selecting the sample of treated teachers, the population was stratified 
between those schools with greater and less than 40 percent minority student body. To 
increase the sample size, 7th grade teachers in high minority schools were added to the 
sample. The randomly selected sample of treated teachers contained 80 mathematics teachers 
and 80 science teachers. To select the control group, regional directors were asked to compile 
lists of teachers, their subject (mathematics or science), grade (7 or 8), and community type 
(city/non-city) who would be participating in 1995 in their region. They were then directed to 
request participation from a specified number of teachers based on subject, grade, and 
community type. In all, the regions were asked to select 68 control teachers m each of 
mathematics and science for participation. 

In mathematics, 46 Discovery teachers returned student tests, for a response rate of 58 
percent while 35 non-Discovery teachers returned student tests, for a response rate ot 51 
percent’ In science, 51 Discovery teachers returned their student tests, for a response rate of 
64 percent while 35 non-Discovery teachers responded, for a response rate of 51 percent. 
Although we recognize the limitations of the research design (including mediocre response 
rates and possible response bias), the results provide useful preliminary information. 



Student Demographics 

From the classes that participated in the study, 1,070 Discovery students and 682 non- 
Discovery students took the mathematics test, and 1,127 Discovery student and 789 non- 
Discovery students took the science test. Table 1 provides demographic information about the 
students who took the mathematics and science Discovery tests. The two samples (students of 
Discovery teachers and students of non-Discovery teachers) are comparable m terms of 
gender, but differ in both ethnicity and community type. In both subjects, a significantly 
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greater proportion of the non-Discovery sample were white, while a greater proportion of the 
Discovery sample were minorities. In the mathematics sample, a significantly larger percent of 
the students of teachers who underwent the Discovery treatment were from rural and urban 
areas while a larger percent of the non-Discovery students were suburban. In the science 
sample, the Discovery group had larger proportions of students from schools in small towns 
and large cities than did the non-Discovery sample. 

The mathematics sample had a slightly larger percent of students of Discovery teachers in 7th 
grade mathematics. The proportion of 8th graders (which includes students responding tot 
they were enrolled in 8th grade mathematics, pre-algebra, and algebra) was similar in both 
samples. There was a greater proportion of students in pre-algebra and algebra classes m the 
mathematics Discovery sample. The science non-Discovery sample contained a larger 
proportion of 8th grade science students, as compared to the Discovery sample. 

Tables 2 and 3 show altitudinal data, and activities of the students who took the mathematics 
and science Discovery test of selected NAEP items. Overall, Discovery and non-Discovery 
students reported similar attitudes towards, and opinions about, mathematics and science. 
However, there were several important and significant differences, m both mathematics and 
science in the activities that students of Discovery teachers experienced. Discovery students 
reported more work in small groups, working with hands-on activities and using language to 
convey both mathematics and science. Discovery students also reported spending less time 
working with textbooks and taking mathematics tests. 

Impact on Student Performance: Analysis of Close-Ended Student Responses 

Given our interest in examining the impact on student performance of teacher participation in 
the Discovery program, and relationships between participation, ^ent gender and ethnicity 
and student performance, we constructed regression models which included teacher 
participation in Discovery, student gender, and student ethnicity as variabl^ofimerest. We 
also included covariates controlling for differences in community types of the samples the 
proportion of students in a school receiving free and reduced unch, and die grade level of 
students. Tables 4 and 5 shows results of these regression analyses for science and 
mathematics, respectively. 

The science model reveals several important findings about the effectiveness of project 
Discovery. First, students of Discovery participants scored significantly higher (t-value 2.7, 
p< Si) than did students of non-participants after controlling for student and conunumty 
factors Second after holding all other things equal, girls performed significantly better than 
bovs (t-value = 2 5 p<05) Third, both white students (t-value = 9.3, p<.001) and students of 
other ethnicities (t-value = 4.8, p<.001) performed significantly bener than black s ^enj Not 
surprisingly, eighth graders outperformed seventh graders (t-value • , P • ) , 

wTa negative axsodation between student performance on the Discovery test and pover* (as 
measured by percentage of students in the school receivmg lunch assistance). Surprising y, 
“ban students signifiLtly outperformed both their small city and suburban counterparts. 

This may be related to the different proportions of students of different ethnicities and from 
different communities in the sample. 
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The mathematics model uncovers some similar details but includes differences as well 
Foremost students of Discovery participants outperformed students of non-part, cipants after 
controlling 5 for student and community variables. However, on the madtemaucs Discovery tesh 
after holdfng constant all other factors (including participation) boys significantly 

alter noiaing T-v,e model also reveals an interaction between 

outperformed girls (t-value P * ^ pi teachers performing significantly 

gender and there weVe no significant differences 

betwen *e 8 'boys of participants and those of non-participants. As on the science te^ both 
between tne ooy f f ethnicities significantly outperformed black students. 

tre^trJ—Ls^^T-r^an smdenteperformed significanUy lower 
than did students from other communities. 

One way to get a bener gra* of die* of 

Discovery test. 

Fiaure 1 which depicts the science performance of the prototypical eighth grade student in a 
£5 «»™ schoo. ’with die 

rS;XTSr^.2^rdi ff ereL, First, on av^ge those 

SdTnts of their black 

teachers. 

Figure 2 shows die mathemadcs perfonnance of die prototypical grade ^em ina 

small town school with the average ^ c " tagc ° 'between students of Discovery 

figure illustrates that the significant differences m i penormoi n - teachers 

mmmm . ss 

their non-Discovery peers. 
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Impact on Student Performance: Analysis of Open-Ended Student Responses 

Each student Discovery XtSon" ^"alllow. 

I^enTof' to taowledge A stratified random sample of 32 classes (eight Discovery, eight 
student of their xn e> . . ^oth^maticst was selected based upon the grade-level 

non-Discovery in each ‘^“j^ of ^ eight regions was sent the open-ended responses 

fr d “f^Tctos (two science, two mathematics). Each region was instructed to convene 
from four classes ttwo scien , , 0 _ en _ ended student response separately, 

teams of science and that scaled the student 

and than tWM* ^ ^ ® r nonsense response) to 4 (excellent response; accurate 

TweTexplainedTAll re Jons completed and returned the mathematics packets, whtle seven 

of the eight science packets were returned. 

~ r-f ssr- 

‘ t; JSSKTr.. 

Discovery students open-ended ques tio 5 b ender m<i ethnicity. In both 

SSSSsiawssassau 

counterparts. 

Conclusion 

• . „ . v . mn | e of how student impact data were collected for a 

Our experience provi P involvement of the larger education community became 

statewide refom effort m whi t th f ^ together a multifaceted group of teachers, 

JJSSS state department of education representatives, university professors, and mdustry- 
project stan, siaie ucpm r ... llted m tests that served multiple purposes. The 

based mathematicians and ** were consistent with the aims of 

mathematics and science test as would meet the needs of the evaluators, 

SSS E-jss »sass- 

ensured that test packete wa endorsement of the testmg process than 

successfully promoting collaboratton among a wide range of stakeholders 

important lessons we have learned through our efforts to measure student impact in the 
context of statewide reform are: 
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We faced numerous difficulties using existing state test data, including. 

1) We could not readily identify and disaggregate treated teachers 
(essentially a unit of analysis problem). 

2) Since there were not teacher identifiers or student identifiers on 
the state database, we would have to access data from each district 
for a specified sample of students (a very labor intensive task). 

3) State test data results were in terms of an overall test score 
(Pass/Fail) rather than individual item results. Such a single score 
is of limited value in understanding student impact. 

Key stakeholders must be intimately involved in the test development 
process to ensure that the instrument is appropriately aligned with the vision 
of the initiative and to provide assistance with test administration and 
processing, especially if there are open-ended or performance task items. 

The approach to student impact data is state specific. Although the Ohio 
SSI has found the NAEP public-release items to provide very useful and 
informative data, our current work with the New Jersey SSI indicates that 
such an approach is not necessarily transferable to other states. Since the 
New Jersey SSI vision includes a very strong commitment to performance- 
based assessment, the use of multiple-choice items is viewed as sending an 
inappropriate message to teachers. Consequently, in an effort to design tests 
that are consistent with the state’s goals, we are in the process of identifying 
performance-based tasks that we would administer to students m New 

Jersey. 
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Table 1 



Sample Sizes and Demographic characteristics of students of mathematics and science teachers 
who attended Discovering and those not yet receiving Discovery training. 



SCIENCE MATHEMATICS 

]Mr>n-Discoverv Discovery Non-DiscQv_e_q Discover 



All Students 



788 1,083 



680 1,065 



Gender 

Female 

Male 



52% 

48% 



56% 54% 55% 

44% 46% 45% 



Race/Ethnicity * 

White 

Black 

Hispanic 

Amer. Indian/ Alaskan Native 
Asian/Pacific Islander 



75% 

19% 

2% 

2 % 

2% 



56% 

32% 

6% 

3% 

2% 



77% 

16% 

2% 

3% 

2% 



67% 

24% 

4% 

3% 

2% 



School Location * 
Rural 

Small Town 
Suburban 
Large City 



28% 


26% 


18% 


37% 


37% 


11% 


18% 


26% 



28% 


51% 


26% 


10% 


36% 


23% 


10% 


16% 



, Significant differences between samples of Discovery and Non-Discovety students in both 
mathematics and science (chi-square p<-05). 
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Table 2 



Types of science classes, classroom activities, and attitudes of students who took the science 
Discovery test. 




Type of Science Class * 

Not taking science this year 
Seventh-grade science 
Eighth-grade science 
Other science class 



<1% 


<1% 


16% 


33% 


79% 


62% 


4% 


4% 



Expected 9th grade science * 


3% 


1% 


Will not take science 


18% 


14% 


General science 


30% 


34% 


Biology or Life Science 


15% 


13% 


Physical Science or Physics 


6% 


10% 


Chemistry 


10% 


3% 


Other science class 
I don’t know 


18% 


27% 


Do the following at least once a week 


59% 


52% 


Problems from textbooks * 


36% 


70% 


Work in small groups * 


21% 


21% 


Use a calculator 


13% 


10% 


Use a computer 
Take science tests 


44% 


44% 



Do the following at all 
Do lab activities * 

Write a few sentences about how solved science 

Make up science problems for others to solve * 
Write reports or do projects * 



74% 

59% 

19% 

55% 



87% 

77% 

24% 

64% 



Agreement with following items 
I like science 
I am good at science 

If I had a choice, I would not study any more science 
I understand most of what goes on in science class 
science is more for boys than girls 
Learning science is mostly memorizing facts 
Almost all people use science in their jobs 
science is useful for solving everyday problems 



60% 

57% 

24% 

74% 

10% 

32% 

49% 

53% 



59% 

54% 

25% 

72% 

7% 

30% 

45% 

52% 




Table 3 



Types of mathematics classes, classroom activities 
mathematics Discovery test. 


, and attitudes of students who took the 




Non- Discovery 


Discovery 


Type of Mathematics Class * 


<1% 


<1% 


Not a mathematics class 


17% 


25% 


Seventh grade math 


51% 


30% 


Eighth grade mathematics 


11% 


23% 


Pre-algebra 


16% 


21% 


Algebra 




1% 


Other mathematics class 


Expected 9th grade mathematics 


i% 


<1% 


Will not take math 


14% 


8% 


Basic, general, business 


13% 


12% 


Pre-Algebra 


28% 


34% 


Algebra I 


18% 


20% 


Geometry 


8% 


6% 


Other mathematics class 


18% 


19% 


I don't know 


Do the following at least once a week 


R2% 


77% 


Problems from textbooks * 


46% 


55% 


Work in small groups * 


69% 


75% 


Use a calculator * 


14% 


14% 


Use a computer 


sn% 


45% 


Take mathematics tests 



Do the following at all 

Work with hands-on manipulatives * 

Write a few sentences about how solved mathematics 

Makeup mathematics problems for others to solve * 
Write reports or do projects * 



71 % 

35% 

24% 

22% 



77% 

51 % 

28% 

40% 



Agreement with following items 
I like mathematics 
I am good at mathematics 

If I had a choice, I would not study any more math 
I understand most of what goes on in math class 
Mathematics is more for boys than girls 
Learning mathematics is mostly memorizing facts 
Almost all people use mathematics in their jobs 
Mathematics is useful for solving everyday problems 



57% 

60% 

21% 

75% 

5% 

44% 

88 % 

79% 



56% 

57% 

19% 

74% 

4% 

41% 

86% 

78% 



• Significant differences between samples of Discovery and Non-Discovery students (chi-square 
p<.05). 
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Table 4 



Full Regression Model showing coefficient and standard errors for student science performance. 
(n= 1144) 



Variable 


Coefficient 
(Standard Error) 


Intercept 


42.27 *** 
(2.42) 


Participation in Discovery 


3.203 ** 
(1.18) 


Percent of Student in Class Receiving 
Free/Reduced Lunch 


JQ *** 

('.03) 


Eighth Grader 


6.10 *** 
(1.30) 


White 


15.44 *** 
(1.65) 


Other Ethnicity 


11.25 *** 
(2.33) 


Female 


2.75 ** 
(1.10) 


Rural 


-2.32 

(1.60) 


Town/Small City 


-4.38 ** 
(1.66) 


Suburb 


-4.22 * 
(1.74) 


D 2 


.19 



Table 4 



Full Regression Model showing coefficients and standard errors for student mathematics 
performance, (n = 1230) 



Variable 


Coefficient 
(Standard Error) 


Intercept 


45.15 *** 
(3.24) 


Participation in Discovery 


1.99 

(1.78) 


Percent of Student in Class Receiving 
Free/Reduced Lunch 


-.21 *** 

(.03) 


Eighth Grader 


3.87 ** 
(1.33) 


White 


11.94 *** 
(1.82) 


Other Ethnicity 


6.07 ** 
(2.32) 


Female 


-5.03 ** 
(1.79) 


Rural 


6.26 ** 
(2.00) 


Town/Small City 


9.97 *** 
(2.10) 


Suburb 


13.36 *** 
(1.92) 


White X Participation Interaction 


4.93 * 
(2.31) 


R 2 


.25 



***p<.001 **p<-01 * p ^ .05 -p^-10 
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Figure 1 



Prototypical plot of predicted science performance for eighth grade 
S the average percentage of students recervtng lunch asststance. 



student in a small town school 




TREATMENT 



a White Female 
O White Male 
■ Black Female 
♦ Black Male 
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Figure 2 



Prototypical plot of predicted mathematics performance for eighth grade student in a small town 
school with the average percentage of students receiving lunch assistance. 




Discovery 



TREATMENT 



O Yhite Male 
□ Yhite Female 
♦ Black Male 
■ Black Female 





Table 5 



Means, standard deviations, and t-tests for Discovery and non-Discovery student responses on 
open-ended science and mathematics questions. 





Discovery 

Mean 

(Stand. Dev) 


Non-Discovery 

Mean 

( Stand. Dev) 


T-Value 


<LL 


SCIENCE 










All Students (n = 224) 


5.79 

(2.78) 


3.83 

(2.99) 


5.05 *** 


224 


Girls (n= 134) 


6.01 

(2.83) 


3.52 

(2.90) 


4.88 *** 


132 


Boys (n = 92) 


5.38 

(2.68) 


4.18 

(3.08) 


2.01 * 


90 


Blacks (n = 81) 


6.42 

(2.88) 


4.18 

(3.24) 


3.29 ** 


79 


Whites (n = 123) 


5.62 

(2.80) 


3.58 

(2.77) 


4.02 *** 


121 


mathematics 










All Students (n = 247) 


4.96 

(2.33) 


3.65 

(2.59) 


4.15 *** 


245 


Girls (n = 151) 


5.08 

(2.14) 


3.51 

(2.61) 


4.07 *** 


149 


Boys (n = 95) 


4.72 

(2.58) 


3.90 

(2.56) 


1.53 


93 


Blacks (n = 48) 


2.80 

(2.39) 


4.46 

(1-81) 


2.27 * 


46 


Whites (n = 184) 


5.02 

(2.39) 


4.02 

(2.59) 


2.61 ** 


182 



*** p < .001 **p<-01 *p<.05 ~p<-10 
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